Derivative-free Optimization Algorithm Research Articles

Convergence Rate of the (1+1)-ES on Locally Strongly Convex and Lipschitz Smooth Functions

Evolution strategy (ES) is one of promising classes of algorithms for black-box continuous optimization. Despite its broad successes in applications, theoretical analysis on the speed of its convergence is limited on convex quadratic functions and their monotonic transformation. In this study, an upper bound and a lower bound of the rate of linear convergence of the (1+1)-ES on locally $L$-strongly convex functions with $U$-Lipschitz continuous gradient are derived as $\exp\left(-\Omega_{d\to\infty}\left(\frac{L}{d\cdot U}\right)\right)$ and $\exp\left(-\frac1d\right)$, respectively. Notably, any prior knowledge on the mathematical properties of the objective function such as Lipschitz constant is not given to the algorithm, whereas the existing analyses of derivative-free optimization algorithms require them.

Simultaneous surrogate modeling and dimension reduction using unsupervised learning. Application to parametric wing shape optimization

This paper presents a machine-learning-based approach that enables simultaneous surrogate modeling and dimension reduction and applies it to aerodynamic parametric shape optimization. Aerodynamic shape optimization is a crucial process in various industries, including aerospace, automotive, and renewable energy. It involves iteratively improving the properties of a system by evaluating an objective function and driving its minimization or maximization using an optimization algorithm. However, the evaluation of aerodynamic objective functions requires computationally expensive operations, such as solving complex fluid dynamics equations and calculating performance metrics like lift and drag coefficients. This computational cost becomes particularly burdensome when derivative-free optimization algorithms need to evaluate numerous samples per iteration. Additionally, when the design space dimension is high, the efficiency and effectiveness of the optimization process decrease. To address these challenges, the paper proposes combining surrogate modeling and dimension reduction. Surrogate modeling constructs a reduced order model that approximates the coefficients of interest in a cost-effective manner, while dimension reduction identifies the most relevant design space dimensions using techniques like Proper Orthogonal Decomposition. The paper suggests an integrative approach that employs Artificial Neural Networks (ANN) and Unsupervised Learning, specifically AutoEncoder networks, to simultaneously build a surrogate model and reduce the problem dimension. This technique is applied to optimize the shape of an airplane wing aerofoil under trans-sonic flight conditions. The wing shape is parameterized using Free Form Deformation (FFD). The paper demonstrates that the suggested approach enables rapid and effective shape optimization.

Benchmarking Derivative-Free Global Optimization Algorithms Under Limited Dimensions and Large Evaluation Budgets

Benchmarking Derivative-Free Global Optimization Algorithms Under Limited Dimensions and Large Evaluation Budgets

Review and Computational Study on Practicality of Derivative-Free DIRECT-Type Methods

Derivative-free DIRECT-type global optimization algorithms are increasingly favoured for their simplicity and effectiveness in addressing real-world optimization challenges. This review examines their practical applications through a systematic analysis of scientific journals and computational studies. In particular, significant challenges in reproducibility have been identified with practical problems. To address this, we conducted an experimental study using practical problems from reputable CEC libraries, comparing DIRECT-type techniques against their state-of-the-art counterparts. Therefore, this study sheds light on current gaps, opportunities, and future prospects for advanced research in this domain, laying the foundation for replicating and expanding the research findings presented herein.

Scaffold Matcher: A CMA-ES based algorithm for identifying hotspot aligned peptidomimetic scaffolds.

The design of protein interaction inhibitors is a promising approach to address aberrant protein interactions that cause disease. One strategy in designing inhibitors is to use peptidomimetic scaffolds that mimic the natural interaction interface. A central challenge in using peptidomimetics as protein interaction inhibitors, however, is determining how best the molecular scaffold aligns to the residues of the interface it is attempting to mimic. Here we present the Scaffold Matcher algorithm that aligns a given molecular scaffold onto hotspot residues from a protein interaction interface. To optimize the degrees of freedom of the molecular scaffold we implement the covariance matrix adaptation evolution strategy (CMA-ES), a state-of-the-art derivative-free optimization algorithm in Rosetta. To evaluate the performance of the CMA-ES, we used 26 peptides from the FlexPepDock Benchmark and compared with three other algorithms in Rosetta, specifically, Rosetta's default minimizer, a Monte Carlo protocol of small backbone perturbations, and a Genetic algorithm. We test the algorithms' performance on their ability to align a molecular scaffold to a series of hotspot residues (i.e., constraints) along native peptides. Of the 4 methods, CMA-ES was able to find the lowest energy conformation for all 26 benchmark peptides. Additionally, as a proof of concept, we apply the Scaffold Match algorithm with CMA-ES to align a peptidomimetic oligooxopiperazine scaffold to the hotspot residues of the substrate of the main protease of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Our implementation of CMA-ES into Rosetta allows for an alternative optimization method to be used on macromolecular modeling problems with rough energy landscapes. Finally, our Scaffold Matcher algorithm allows for the identification of initial conformations of interaction inhibitors that can be further designed and optimized as high-affinity reagents.

Multi-parametric optimization of magnetic resonance imaging sequences for magnetic resonance-guided radiotherapy

Background and purposeMagnetic Resonance Imaging (MRI) is widely used in oncology for tumor staging, treatment response assessment, and radiation therapy (RT) planning. This study proposes a framework for automatic optimization of MRI sequences based on pulse sequence parameter sets (SPS) that are directly applied on the scanner, for application in RT planning. Materials and methodsA phantom with seven in-house fabricated contrasts was used for measurements. The proposed framework employed a derivative-free optimization algorithm to repeatedly update and execute a parametrized sequence on the MR scanner to acquire new data. In each iteration, the mean-square error was calculated based on the clinical application. Two clinically relevant optimization goals were pursued: achieving the same signal and therefore contrast as in a target image, and maximizing the signal difference (contrast) between specified tissue types. The framework was evaluated using two optimization methods: a covariance matrix adaptation evolution strategy (CMA-ES) and a genetic algorithm (GA). ResultsThe obtained results demonstrated the potential of the proposed framework for automatic optimization of MRI sequences. Both CMA-ES and GA methods showed promising results in achieving the two optimization goals, however, CMA-ES converged much faster as compared to GA. ConclusionsThe proposed framework enables for automatic optimization of MRI sequences based on SPS that are directly applied on the scanner and it may be used to enhance the quality of MRI images for dedicated applications in MR-guided RT.

Risk‐based optimization of concentrically braced tall timber buildings: Derivative free optimization algorithm

AbstractMass timber materials are attractive alternatives for tall‐timber buildings (TBs), where the need for sustainability is apparent. Innovative structural systems and design methodologies are needed to fulfil performance requirements according to modern performance based approaches. This paper deals with the design and optimization of buckling restrained braces as earthquake protection system for tall‐TBs through risk‐based design procedure. This procedure controls the mean annual frequency of exceedance of several limit states evaluated through a SAC‐FEMA approach and using response spectrum linear analyses on linearized models for demand assessment. The features of the optimization procedure and the linearized models are shown through an application on a 20‐story mass‐TB located in a high seismic zone. The optimization is executed through a derivative‐free algorithm, the generalized pattern Search, adopting several solution strategies whose efficiency and effectiveness for this kind of applications are shown and discussed. Finally, the results are compared and validated through the execution of non‐linear analyses within a multiple stripe framework.

Experimental analysis and optimization of abrasive waterjet deep hole drilling process parameters for SS AISI 316L

Recent breakthroughs in component downsizing and miniaturization emphasise the requirement for deep-hole drilling with an increased aspect ratio, especially in the automobile sector for fuel injectors and the medical sector for manufacturing bone screws or surgical equipment. In this research, deep hole drilling on stainless steel AISI 316L has been conducted using an Abrasive Waterjet Machine (AWJM), and the influence of drilling variables on the material removal rate, roundness deviation of drilled holes have been evaluated to assess the machining and hole characteristics. In addition, machining parameters' statistical relevance has been investigated using a multi-parametric analysis of variance. Quadratic mathematical models for material removal rate and roundness deviation have been established by correlating drilling parameters. The Grey Wolf Optimization (GWO) algorithm has been applied in this research to identify the optimal combination of deep hole drilling parameters for maximizing material removal rate and minimizing the roundness deviation. The results have been compared with Derivative-free optimization, Whale optimization, and Harmony search algorithms. The comparison revealed that the GWO algorithm performed better than other algorithms. In addition, a validation test has been carried out to confirm the accuracy of the results produced by the GWO. The images obtained from the scanning electron microscope showed that the surfaces of the deep holes are smooth. Additionally, the ploughing action was shown to be the principal mechanism responsible for removing the material.

A Derivative-Free Optimization Algorithm Combining Line-Search and Trust-Region Techniques

A Derivative-Free Optimization Algorithm Combining Line-Search and Trust-Region Techniques

Comparison of Derivative-Free Optimization: Energy Optimization of Steam Methane Reforming Process

In modern chemical engineering, various derivative-free optimization (DFO) studies have been conducted to identify operating conditions that maximize energy efficiency for efficient operation of processes. Although DFO algorithm selection is an essential task that leads to successful designs, it is a nonintuitive task because of the uncertain performance of the algorithms. In particular, when the system evaluation cost or computational load is high (e.g., density functional theory and computational fluid dynamics), selecting an algorithm that quickly converges to the near-global optimum at the early stage of optimization is more important. In this study, we compare the optimization performance in the early stage of 12 algorithms. The performance of deterministic global search algorithms, global model-based search algorithms, metaheuristic algorithms, and Bayesian optimization is compared by applying benchmark problems and analyzed based on the problem types and number of variables. Furthermore, we apply all algorithms to the energy process optimization that maximizes the thermal efficiency of the steam methane reforming (SMR) process for hydrogen production. In this application, we have identified a hidden constraint based on real-world operations, and we are addressing it by using a penalty function. Bayesian optimizations explore the design space most efficiently by training infeasible regions. As a result, we have observed a substantial improvement in thermal efficiency of 12.9% compared to the base case and 7% improvement when compared to the lowest performing algorithm.

Comparing procedures for estimating random coefficient logit demand models with a special focus on obtaining global optima

We compare several nested fixed point and optimization procedures for computing the estimator of the widely-used empirical market demand model developed by Berry et al. (1995). It is well-known that the optimization may often lead to multiple local optima, which, if ignored, can lead to erroneous policy conclusions. By combining the frequencies of finding the global minima and the computing times, we propose a new indicator that provides the computing time needed for obtaining the global minima. Using this indicator, we find that the Spectral and Squarem methods (Reynaerts et al., 2012) outperform the benchmark contraction iterations method and the MPEC (Dubé et al., 2012) and ABLP (Lee and Seo, 2015) methods. Moreover, when the share of the outside alternative is relatively large, two derivative-free optimization algorithms, which require less calculations and coding than derivative-based algorithms, outperform the best derivative-based methods. A simple argument suggests that the latter statement is likely to be true for other versions of the model as well.

Hybrid data-driven framework for shale gas production performance analysis via game theory, machine learning, and optimization approaches

A comprehensive and precise analysis of shale gas production performance is crucial for evaluating resource potential, designing a field development plan, and making investment decisions. However, quantitative analysis can be challenging because production performance is dominated by the complex interaction among a series of geological and engineering factors. In fact, each factor can be viewed as a player who makes cooperative contributions to the production payoff within the constraints of physical laws and models. Inspired by the idea, we propose a hybrid data-driven analysis framework in this study, where the contributions of dominant factors are quantitatively evaluated, the productions are precisely forecasted, and the development optimization suggestions are comprehensively generated. More specifically, game theory and machine learning models are coupled to determine the dominating geological and engineering factors. The Shapley value with definite physical meaning is employed to quantitatively measure the effects of individual factors. A multi-model-fused stacked model is trained for production forecast, which provides the basis for derivative-free optimization algorithms to optimize the development plan. The complete workflow is validated with actual production data collected from the Fuling shale gas field, Sichuan Basin, China. The validation results show that the proposed procedure can draw rigorous conclusions with quantified evidence and thereby provide specific and reliable suggestions for development plan optimization. Comparing with traditional and experience-based approaches, the hybrid data-driven procedure is advanced in terms of both efficiency and accuracy.

Operations research contribution to totally automated radiotherapy treatment planning: A noncoplanar beam angle and fluence map optimization engine based on optimization models and algorithms

The planning of radiotherapy treatments is based on the use of mathematical optimization models and algorithms. A treatment plan will correspond to an admissible solution to a problem that is defined by the medical prescription. The medical prescription defines the constraints that have to be satisfied, and that are patient dependent. To find a treatment plan complying with all the defined constraints, an objective function is built, having a set of parameters that can be tuned. In the clinical practice, the objective function parameters are tuned through a trial-and-error procedure. One common way of defining this objective function is to consider as parameters weights that are related with the importance of the corresponding structures of interest (volumes to treat and organs to spare), as well as upper and lower bounds that are related with the constraints defined by the medical prescription. Some treatment options, like the set of irradiation directions (beam angles), are usually defined beforehand by the planner, considering previous experiences with similar cases. This trial-and-error process is lengthy, since it can take up to several hours to calculate an admissible treatment plan for a single patient. There have been some research efforts to automate the treatment planning procedure, releasing the planner for other important tasks in the radiotherapy treatment workflow and guaranteeing the consistent calculation of high-quality plans. In this work derivative-free optimization algorithms are integrated with fuzzy inference systems that automatically tune the objective function parameters so that an admissible solution is found. This fully automated approach was tested and assessed considering six head-and-neck cancer cases already treated at the Portuguese Institute of Oncology at Coimbra (IPOC). For the clinical cases tested, comparisons between different treatment plans clearly favor the proposed approach. For a similar tumor coverage, it was possible to improve the sparing of the spinal cord, brainstem and parotids. Automating radiotherapy treatment planning can contribute to improved treatment plans in a consistent way.

Multi-criterion optimization of invasive antenna applicators for Au@Fe3O4, Au@-Fe2O3 and Au@-Fe2O3 mediated microwave ablation treatment

ABSTRACT Magnetic nanoparticle (MNP) mediated microwave ablation has the great potential at present to address challenges associated with treatment planning such as maximum heat generation in the vicinity of targeted tissues in lesser penetration time. Further, the antenna applicators injected in human phantom must be rigid and thin. The derivative-free optimization algorithms are carried out for optimum design of monopole, slot, dipole, and tapered slot antenna applicators for ablation of tumour tissues invasively. It is found that in terms of input impedance matching, the used multi-criterion Nelder-Mead optimization performs efficiently for tapered slot applicator achieving S 11 value of –40 dB with much reduced antenna dimensions. In order to further escalate the performance of tapered slot antenna, gold (Au)-coated iron-based MNPs are suggested for tumor infusion. Spherical gold-coated shell material is preferrable for more sphericity of ablation zone, biocompatibility and due to high conductivity, heat generated in MNPs can be transferred to biological tissues more rapidly. The size, type, and shape of MNPs also influence the heat generation in tumor tissues. Thus, three different types of MNPs having high magnetization properties, Au@Fe3O4, Au@-Fe2O3 and Au@-Fe2O3 have been employed to study the performance in terms of maximum rise in temperature, specific absorption rate (SAR), and area of ablation zone by varying core size radius of MNPs. Results demonstrate that increase in radius of MNP core helps in increasing the temperature distribution and reduction in ablation zone. The optimized lesion is achieved for 20 nm core radius of Au@Fe3O4.

Derivative-Free Optimization With Proxy Models for Oil Production Platforms Sharing a Subsea Gas Network

The deployment of offshore platforms for the extraction of oil and gas from subsea reservoirs presents unique challenges, particularly when multiple platforms are connected by a subsea gas network. In the Santos basin, the aim is to maximize oil production while maintaining safe and sustainable levels of CO2 content and pressure in the gas stream. To address these challenges, a novel methodology has been proposed that uses boundary conditions to coordinate the use of shared resources among the platforms. This approach decouples the optimization of oil production in platforms from the coordination of shared resources, allowing for more efficient and effective operation of the offshore oilfield. In addition to this methodology, a fast and accurate proxy model has been developed for gas pipeline networks. This model allows for efficient optimization of the gas flow through the network, taking into account the physical and operational constraints of the system. In experiments, the use of the proposed proxy model in tandem with derivative-free optimization algorithms resulted in an average error of less than 5% in pressure calculations, and a processing time that was over up to 1000 times faster than the phenomenological simulator. These results demonstrate the effectiveness and efficiency of the proposed methodology in optimizing oil production in offshore platforms connected by a subsea gas network, while maintaining safe and sustainable levels of CO2 content and pressure in the gas stream.

Error Analysis of Surrogate Models Constructed Through Operations on Submodels

Model-based methods are popular in derivative-free optimization (DFO). In most of them, a single model function is built to approximate the objective function. This is generally based on the assumption that the objective function is one black box. However, some real-life and theoretical problems show that the objective function may consist of several black boxes. In those problems, the information provided by each black box may not be equal. In this situation, one can build multiple submodels that are then combined to become a final model. In this paper, we analyze the relation between the accuracy of those submodels and the model constructed through their operations. We develop a broad framework that can be used as a theoretical tool in model error analysis and future research in DFO algorithm design. Funding: Y. Chen’s research is partially funded by the MITACS Globalink program. All authors research partially supported by NSERC of Canada Discovery [Grant 2018-03865].

Gradient Approximation and Multivariable Derivative-Free Optimization Based on Noncommutative Maps

In this article, multivariable derivative-free optimization algorithms for unconstrained optimization problems are developed. A novel procedure for approximating the gradient of multivariable objective functions based on noncommutative maps is introduced. The procedure is based on the construction of an exploration sequence to specify where the objective function is evaluated and the definition of so-called gradient generating functions which are composed with the objective function, such that the procedure mimics a gradient descent algorithm. Various theoretical properties of the proposed class of algorithms are investigated and numerical examples are presented.

Global linear convergence of evolution strategies with recombination on scaling-invariant functions

Evolution Strategies (ESs) are stochastic derivative-free optimization algorithms whose most prominent representative, the CMA-ES algorithm, is widely used to solve difficult numerical optimization problems. We provide the first rigorous investigation of the linear convergence of step-size adaptive ESs involving a population and recombination, two ingredients crucially important in practice to be robust to local irregularities or multimodality. We investigate the convergence of step-size adaptive ESs with weighted recombination on composites of strictly increasing functions with continuously differentiable scaling-invariant functions with a global optimum. This function class includes functions with non-convex sublevel sets and discontinuous functions. We prove the existence of a constant r such that the logarithm of the distance to the optimum divided by the number of iterations converges to r. The constant is given as an expectation with respect to the stationary distribution of a Markov chain—its sign allows to infer linear convergence or divergence of the ES and is found numerically. Our main condition for convergence is the increase of the expected log step-size on linear functions. In contrast to previous results, our condition is equivalent to the almost sure geometric divergence of the step-size on linear functions.

Comparison of parallel optimization algorithms on computationally expensive groundwater remediation designs

Contaminated groundwater resources threaten human health and destroy ecosystems in many areas worldwide. Groundwater remediation is crucial for environmental recovery; however, it can be cost prohibitive. Planning a cost-effective remediation design can take a long time, as it may involve the evaluation of many management decisions, and the corresponding simulation models are computationally demanding. Parallel optimization can facilitate much faster management decisions for cost-effective groundwater remediation design using complex pollutant transport models. However, the efficiency of different parallel optimization algorithms varies depending on both the search strategy and parallelism. In this paper, we show the performance of a parallel surrogate-based optimization algorithm called parallel stochastic radial basis function (p-SRBF), which has not been previously used on contaminant remediation problems. The two case problems involve two superfund sites (i.e., the Umatilla Aquifer and Blaine Aquifer), and one objective evaluation takes 5 and 30 min for the two problems, respectively. Exceptional speedup (superlinear) is achieved with 4 to 16 cores, and excellent speedup is achieved using up to 64 cores, obtaining a good solution at 80 % efficiency. We compare our p-SRBF with three different parallel derivative-free optimization algorithms, including genetic algorithm, mesh adaptive direct search, and asynchronous parallel pattern search optimization, in terms of objective quality, computational reduction and robust behavior across multiple trials. p-SRBF outperforms the other algorithms, as it finds the best solution in both the Umatilla and Blaine cases and reduces the computational budget by at least 50 % in both problems. Additionally, statistical comparisons show that the p-SRBF results are better than those of the alternative algorithms at the 5 % significant level. This study enriches theoretical real-world groundwater remediation methods. The results demonstrate that p-SRBF is promising for environmental management problems that involve computationally expensive models.

A subset-selection-based derivative-free optimization algorithm for dynamic operation optimization in a steel-making process

ABSTRACT Steel-making production is a dynamic process that has the characteristics of high temperature and heat, and a complex reaction mechanism that causes the mechanism model possibly to be unavailable and the system to be a black box. In this article, a dynamic operation optimization (DOO) problem is refined from the basic oxygen furnace (BOF) steel-making process, and the system model is formulated by a data analytics method. This prevents to solve the optimization problem with derivative-based optimization methods. To circumvent these difficulties, a surrogate-model-based derivative-free optimization algorithm is proposed for solving the DOO problem. In order to establish the surrogate model with the least number of function evaluations, a subset selection strategy is designed to find a sparse structure for the optimization problem, based on which a set of simple bases is determined to establish the surrogate model. Moreover, this also reduces the scale of the parameter optimization problem. Numerical experiments on actual production data verify the applicability and effectiveness of the proposed method.